Point in Time Remote Copy for Multiple Sites

ABSTRACT

A method for copying data to multiple remote sites includes transmitting data from a first volume in a primary storage system to a back-up volume provided in a secondary storage system. The primary storage system is located at a primary site, and the secondary storage system is located at a first remote site from the primary site. The data from the first volume in the primary storage system is copied to a second volume in the primary storage system using a point in time (PiT) as a reference point of time for the copying. The second volume is provided with a first time consistent image of the first volume with respect to the reference point of time. The data from the second volume in the primary storage system is transferred to a third volume in a ternary storage system at a second remote site. The third volume is provided with a second time consistent image of the second volume with respect to the reference point of time, where the second time consistent image is a mirror image of the first time consistent image. The data from the third volume is transferred to a fourth volume in the ternary storage system. The fourth volume is provided with a third time consistent image. In the ternary storage, either of the third volume or fourth volume can always keep time consistent image of the first volume.

BACKGROUND OF THE INVENTION

The present application relates to a data storage system, moreparticularly to a storage system for performing remote copies tomultiple sites.

Industry and commerce have become so dependent on computer systems withonline or interactive applications that an interruption of only a fewminutes in the availability of those applications can have seriousfinancial consequences. Outages of more than a few hours can sometimesthreaten a company's or an institution's existence. In some cases,regulatory requirements can impose fines or other penalties fordisruptions or delays in services that are caused by applicationoutages.

As a consequence of this growing intolerance for application outages,there is a keen interest in improving the availability of theseapplications during normal operations and in decreasing the amount oftime needed to recover from equipment failure or other disastroussituations.

In addition to these brief interruptions in normal operations,precautions must be taken for longer-duration outages caused bydisasters such as equipment or software failure, fire, flood,earthquake, airplane crashes, terrorist or vandal activitiesRealistically, these outages cannot be avoided entirely but theprobability of an extended outage can be reduced to an arbitrarily smallvalue by implementing complex systems of geographically dispersedcomponents with redundant features that have no single point of failure.

The exposure to an extended outage can be mitigated by providing asecondary site for storing redundant or back-up data, as disclosed inU.S. Pat. No. 6,539,462, which is incorporated by reference. The databack-up can be made by performing remote data copy or local data copy(shadow copy) methods. Examples of remote copy and shadow copy are“Truecopy” and “ShadowImage” performed by some of Hitachi storagedevices.

Generally a data storage center includes a primary storage site and asecondary storage site. The secondary storage site keeps a copy ofproduction data of primary storage systems at a primary site to increaseredundancy. The primary site serves as a production site and containsprimary hosts and primary storage systems, and the secondly site servesas a standby and contains secondary hosts and secondary storage systems.The production data are copied from the primary storage systems to thesecondly storage systems.

Recently, more and more businesses need to copy data to multiple storagesystems at multiple sites to enhance redundancy and data availability.Some businesses are afraid of losing data kept in not only the primarysite but also that stored in the secondary site at the same time,particularly if the business has both primary and secondary storagesystems within close proximity of each other, e.g., within the samebuilding. Accordingly, some businesses are interested in having a datastorage system that has three or more storage sites for multiple dataredundancies at these sites to avoid permanent loss of data as a resultof a regional disaster.

In addition to their use in disaster recovery, the data redundancies maybe used to provide greater data availability to users, e.g., global datasharing. Businesses or enterprises have several sites that aregeographically separated that perform separate information processingoperations. These enterprises, however, may wish to share data betweenthese multiple sites. For example, master data of products informationare managed and updated at the headquarter site, which are distributedto local sites for read only access, thereby providing quicker dataaccess to users. Accordingly, global data sharing is commonly used forenterprise data warehousing.

One type of remote copy technology is a point in time (PiT) remote copytechnology that enables users to have a point in time data copy of theprimary site at the secondary site. That is, the data stored at theprimary site is copied to the secondary site, so that the secondary sitemirrors the data content of the primary site at a given instant of time.

The PiT technology is useful for providing data consistency or timeconsistent data among multiple storage systems at the secondary site.The requirement of keeping data consistency among multiple storagesystems is common in disaster recovery for both a very large applicationthat issue input and output requests (IOs) to several storage systems atthe same time and a group of applications related to each other thatrequires data consistency among each of data applications. The PiTtechnology is also used for global data sharing, which requires dataconsistency among multiple storage systems, so that distributedapplications may refer to the time consistent image.

However, conventional remote copy technologies are designed to provideremote copy between only two sites. Even if these technologies areextended to support three or more sites, it is difficult to keep dataconsistency among multiple storage systems without a new technology.

BRIEF SUMMARY OF THE INVENTION

In one embodiment, a method for copying data to multiple remote sitesincludes transmitting data from a first volume in a primary storagesystem to a back-up volume provided in a secondary storage system. Theprimary storage system is located at a primary site, and the secondarystorage system is located at a first remote site from the primary site.The data from the first volume in the primary storage system is copiedto a second volume in the primary storage system using a point in time(PiT) as a reference point of time for the copying. The second volume isprovided with a first time consistent image of the first volume withrespect to the reference point of time. The data from the second volumein the primary storage system is transferred to a third volume in aternary storage system at a second remote site. The third volume isprovided with a second time consistent image of the second volume withrespect to the reference point of time.

The data from the third volume in the ternary storage system is copiedto a fourth volume in the ternary storage system. The fourth volume isprovided with a third time consistent image corresponding to the secondtime consistent image. The third time consistent image is substantiallythe same as the first time consistent image. The transmitting stepinvolves a synchronous remote copying method, and the transferring stepinvolves an asynchronous remote copying method. The first and secondtime consistent images are substantially the same.

A plurality of data write requests is received at the primary storagesystem from a primary host. Each of the data write requests has atimestamp and data associated with that write request. The data writerequests are stored in the primary storage system. The copying stepincludes retrieving first timestamp associated with first data; andshadow copying the first data from the first volume to the second volumeif the first timestamp indicates a time that is prior to the referencepoint of time. The method also includes retrieving second timestampassociated with the second data; and not copying the second data to thesecond volume if the second timestamp indicates a time that issubsequent to the reference point of time. The method also includessuspending the copying step if all data stored in the first volumehaving timestamps that precede the reference point of time are shadowcopied to the second volume.

The method also includes copying the data from the third volume in theternary storage system to a fourth volume in the ternary storage system,the fourth volume being provided with a third time consistent imagecorresponding to the second time consistent image; and estimating timerequired for providing the fourth volume with the third time consistentimage using information relating to the an amount of data copied fromthe first volume to the second volume in a previous copy cycle. Theamount of data corresponds to the data copied to the second volume fromthe first volume to provide the second volume with the second timeconsistent image. The information relating to the amount of data copiedfrom the first volume to the second volume corresponds to a copy timeneeded to provide the second volume with the second time consistentimage.

In another embodiment, a method for copying data to a remote siteincludes copying data from a first volume to a second volume to providethe second volume with a first time consistent image with respect to afirst given time. The first and second volumes are provided in a firststorage system. The data from second volume is transferred to a thirdvolume to provide the third volume with a second time consistent imagewith respect to a second given time. The third volume is provided in asecond storage system that is located at least 10 miles from the firststorage system. The data stored in the first volume is transmitted to aback-up volume provided in a third storage system. The transmitting stepinvolves a synchronous remote copying method and the transferring stepinvolves an asynchronous remote copying method.

The method also includes receiving the data at the first storage systemfrom a third storage system. The receiving step involves a synchronousremote copying method and the transferring step involves an asynchronousremote copying method. The first and second storage systems are storagesub-systems. The first given time and the second given time are thesame. The copying step includes copying the data from the first volumeto a plurality of secondary volumes to provide each of the secondaryvolumes with the first time consistent image with respect to the firstgiven time. The first and secondary volumes are provided in the firststorage system. The transferring step includes transferring the datafrom the secondary volume to a plurality of ternary volumes to provideeach of the ternary volumes with the second time consistent image withrespect to the second given time. The ternary volumes are provided in aplurality of secondary storage systems that are located at least 10miles from the first storage system.

In another embodiment, a computer system includes a timer to provide atimestamp to data requests; an interface configured to form acommunication link with a first storage sub-system; and a computerstorage medium. The medium includes code for initiating copying of datafrom a first volume to a second volume to provide the second volume witha first time consistent image with respect to a first given time, thefirst and second volumes being provided in a first storage sub-system,and code for initiating transferring of the data from second volume to athird volume to provide the third volume with a second time consistentimage with respect to a second given time, the third volume beingprovided in a second storage system that is located at least 10 milesfrom the first storage sub-system. The computer system is a host coupledto the first storage sub-system or a storage system including aplurality of storage sub-systems provided at multiple sites.

In yet another embodiment, a computer readable medium for use in astorage system includes code for copying data from a first volume to asecond volume to provide the second volume with a first time consistentimage with respect to a first given time, the first and second volumesbeing provided in a first storage system; and code for transferring thedata from second volume to a third volume to provide the third volumewith a second time consistent image with respect to a second given time,the third volume being provided in a second storage system that islocated at least 10 miles from the first storage system. The medium isprovided in a host or storage sub-system.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an information processing system including a primary site,a secondary site, and a ternary site and configured for remote copyingto multiple sites according to one embodiment of the present invention.

FIG. 2 illustrates a block diagram of a system that is configured toprovide the PiT shadow copy according to one embodiment of the presentinvention.

FIG. 3 illustrates a process for performing a PiT shadow copy from theA-Vol to the B-Vol to form a volume pair according to one embodiment ofthe present invention

FIG. 4 illustrates a state transition of a PiT remote copy processassociated with a PiT shadow copy process according to one embodiment ofthe present invention.

FIG. 5 shows a process performed by the control program during the PiTremote copy and PiT shadow copy processes according to one embodiment ofthe present invention.

FIG. 6 illustrates an information processing system configured forremote copying to multiple sites according to another embodiment of thepresent invention.

FIG. 7 shows a block diagram an information processing system configuredfor remote copying to multiple sites according to one embodiment of thepresent invention.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the present invention are directed to providing remotecopying to three or more sites. A storage system according to anembodiment of the present invention provides more reliable disasterrecovery solution since data may be copied to three of more sites. Sucha storage system also enables an effective global data sharing solutionsince it enables data to be shared or distributed to multiplegeographical sites.

FIG. 1 shows an information processing system (or a storage system) 500including a primary site 1, a secondary site 2, and a ternary site 3.Each site includes one or more hosts 10 (not shown for secondary andternary sites). Each site also includes one or more storage systems 20a-h, 23 a-h, and 27 a-h. These storage systems may also be referred toas storage sub-systems and include disk array units. The hosts andstorage systems are connected to each other for IOs 15 a-h via one ormore communication links 13 a-h. Examples of the communication linksinclude ESCON, FICON and FibreChannel. The host 10 of the primary site 1includes a timer II to provide time-stamps 16 a-h to the IOs 15 a-hdirected to the storage systems 20 a-h. Each timer in the hosts issynchronized with the others to provide the same time for respectiveIOs.

The primary site 1 and the secondary site 2 is coupled to each other viaa wide area network according to one embodiment of the presentinvention. The primary site 1 and the ternary site 3 are coupled to eachother via a wide area network. Examples of the wide area network areDWDM, ATM, SONET, and Internet. Data paths or links 50 a-h are logicalremote copy paths between the storage system in the primary site and thestorage system in the secondary site. Data paths or links 57 a-h arelogical remote copy lines between the storage system in the primary siteand the storage system in the ternary site.

In the primary site 1, the storage systems 20 a-h contain two types ofvolumes, A-Vols 30 a-h and B-Vols 31 a-h. The B-Vols 31 a-h are theshadow copy targets of A-Vols 30 a-h. The data paths 40 a-h indicatelogical links of shadow copy between A-Vols 30 a-h and B-Vols 31 a-h inthe same storage systems 20 a-h.

In the secondary site 2, the storage systems 23 a-h contain volumesS-Vols 33 a-h, which are the synchronous remote copy targets of A-Vols31 a-h. Data paths 50 a-h indicate logical links of remote copy betweenA-Vols 30 a-h and S-Vols 33 a-h.

In the ternary site 3, the storage systems 27 a-h includes two types ofvolumes, C-Vols 37 a-h and D-Vols 38 a-h. The C-Vols 37 a-h are theasynchronous remote copy targets of B-Vols 31 a-h, and the D-Vols 38 a-hare the shadow copy targets of C-Vols 37 a-h. Data paths 57 a-h indicatelogical links of remote copy between B-Vols 31 a-h and C-Vols 37 a-h.Data paths 47 a-h indicate logical links of shadow copy between C-Vols37 a-h and D-Vols 38 a-h in the same storage systems 27 a-h.

In operation, the primary site 1 serves as a production site thatgenerates production data in its storage systems. The secondary site 2serves as a “hot or warm” standby site, which takes over the operationfrom the primary site and continues the production automatically orsemi-automatically in case of planned shutdowns or unplanned outages.The planned shutdowns include system upgrades and maintenances. Theunplanned outages include system failures resulting from device failuresor natural disaster.

For such situations, the secondary site 2 keeps synchronous remote copydata in its storage systems. Generally, synchronous techniques involvereceiving write commands from a host and confirming the successfulrecording of new information on the primary and secondary storagesystems before acknowledging to the host that the write command has beencompleted.

Accordingly, the primary and secondary sites are generally locatedrelatively close to each other, e.g., within the same city, metropolitanregion, campus, or building, to prevent data transmission delaysresulting from the synchronous remote copy procedure. In one embodiment,the primary site and the secondary site are provided within about 10miles of each other. In another embodiment, the two sites are providedwithin about 100 miles of each other.

The ternary site 3 operates as a “cold” standby site, which takes overthe operation from the primary site 1 and continues the production incase of planned shutdowns or unplanned outages. The ternary site mayalso serve as a new primary site if a region disaster occurs and takesboth the primary and secondary sites 1 and 2 off the system. The ternarysite 3 receives data from the primary site via asynchronous remote copymethods according to one embodiment of the present invention.Accordingly, the ternary site 3 may be located at any distance from theprimary site, either far or close. In one embodiment, the ternary siteis located more than 10 miles from the primary site. In anotherembodiment, the ternary site is located more than 100 miles from theprimary site.

The remote copy and shadow copy operations are controlled by a controlprogram 12 provided in the host 10 according to one embodiment of thepresent invention.

The information processing system 500 provides enhanced dataavailability and disaster recovery. For example, if a minor disastercauses the primary site 1 to fail, the operation will fail over to thesecondary site 2. If a major disaster brings down both the primary site1 and the secondary site 2, operations of these sites can fail over tothe ternary site 3. The possibility of losing all three sites at thesame time is very remote.

In one embodiment, the primary storage systems 20 a-20 h use asynchronous remote copy method and a combination of shadow copy andasynchronous remote copy. One volume may be provided with a remote copylink and a shadow copy link at the same time because each operationessentially has the different logics and does not require complicatedstate transition mechanisms. In addition, the volume that is the targetof the shadow copy link may be provided with a remote copy link to avolume in the remote storage system.

The above approach solves the problem of providing the primary storagesystems 20 a-h with both synchronous remote copy links 50 a-h to thesecondary storage systems 23 a-h and asynchronous remote copy links 57a-h to the ternary storage systems 27 a-h at the same time, therebyavoiding the difficulty and high cost of providing a volume with complexstate transition mechanisms for multiple remote copy links at the same.

Another issue associated with the information processing system 500having three or more data centers is the need to maintain dataconsistency or time consistent data at these sites. In one embodiment,such a condition is needed since the IOs or data requests from one hostor several hosts are provided to all these storage sites. The ternarystorage systems 27 a-h receive asynchronous remote copy from the primarystorage systems 20 a-h. Generally, asynchronous remote copy does notassure data consistency between multiple storage systems withoutquiescing the application.

One method of resolving the above problem involves the use of PiT shadowcopy method, as explained below. The PiT copying provides a copy ofinformation that is consistent at some prescribed point in time. The PiTshadow copy provides a shadow copy of data stored in A-Vol 30 at aspecific point in time to B-Vol 31.

FIG. 2 illustrates a block diagram of a system 502 that is configured toprovide the PiT shadow copy according to one embodiment of the presentinvention. The system 502 corresponds to the primary site 1 of FIG. 1.

The storage system 20, e.g., disk array unit, includes one or morecontrollers 21 to process the data requests (IOs) from the host 10 aswell as other administrative operations of the storage system 20, e.g.,remote copy, shadow copy, and so on. One or more cache 22 is provided totemporarily store data that are frequently accessed as well as controldata for shadow and remote copies.

A communication link 41 couples the controller 21 to the cache 22. Acommunication link 42 couples the controller 21 to volumes 30 and 31.Examples of the communication links are PCI, FibreChannel, and the like.A data path 40 indicates a logical link of shadow copy from A-Vol 30 toB-Vol 31

The controller includes a PiT Freeze 100 to facilitate the PiT copyprocess. When a user or the system executes this function and inputs agiven time, the controller 21 prevent write requests (or IOs) with thetime-stamp after that the given time from being copied in shadow copyfrom A-Vol 30 to B-Vol 31, as explained below in connection with FIG. 3.A volume pair, consistency group, or other volume pair group is formedas a result of the following operation.

FIG. 3 illustrates a process for performing a PiT Freeze commandaccording to one embodiment of the present invention. At step 101, thetimestamp of each data being replicated from A-Vol 30 to B-Vol 31 ischecked. The timestamp 16 is added to the IOs 15 to be written to thestorage system by the timer 11 of the host (see FIGS. 1 and 2). If thereare a plurality of hosts in the primary site, each host is provided witha timer that is synchronized with each other. It is determined whetheror not the timestamp associated with the data being examined indicates atime that is before or after the specified time (or PiT). If thetimestamp is before the PiT, the data associated with the timestamp iscopied to the B-Vol (step 103). Thereafter, next data is retrieved tocontinue the copy process.

In another embodiment, all IOs to storage systems are frozen, and thandummy IOs (or Mark IOs) are issued to keep time consistency among thestorage systems. The process detects the dummy IOs instead of theparticular timestamps when it operates PiT Freeze.

If the timestamp is after the PiT, the shadow copy process is suspended(step 104). As a result, the B-Vol mirrors the data content of the A-Volas of the specified PiT. The contents of the B-Vol is then copied to thestorage system in the ternary site 3, as explained below. The controller21 sets the mode of each volume as the Freeze Mode to indicate B-Vol 31contains the specific point in time shadow copy of A-Vol 30 (step 105).

In one embodiment, each of the IOs from the host 10 is also providedwith a sequence number to indicate its order relative to other IOs. Theshadow copy process of FIG. 3 checks the sequence numbers of the IOs toensure that the Freeze mode is set after all IOs having timestampsbefore the PiT have been copied to the B-Vol. The use of sequence numberensures the integrity of the PiT shadow copy process to the targetvolume (B-Vol) even if some IOs are written to the storage system outsequence as a result of a bottle neck at a given point in the networkconnection. In one implementation, both the sequence number and thetimestamps are checked for the copy process. In another implementation,the sequence number is used instead of the timestamp.

Referring back to FIG. 2, the controller 21 also includes a PiT Query120 operable to check the mode of each volume or volume group and returnthe result. The modes include a Freeze Mode or non-Freeze Mode. In oneembodiment, the volume group is a Consistency Group. The ConsistencyGroup includes a plurality of volume pairs that keep 10 consistencywithin the group. The Query 120 informs the control program 12 in thehost 10 as to the status of the shadow copy link between A-Vol 30 a-hand B-Vol 31 a-h. Issuing a PiT Query command for a Consistency Group ismore efficient than issuing commands for all volume pairs in the system.

FIG. 4 illustrates a state transition of a PiT remote copy processassociated with a PiT shadow copy process according to one embodiment ofthe present invention. The storage systems 300 and 301 correspond to thestorage systems 20 a-h and 27 a-h in FIG. 1, respectively. That is, thestorage system 300 represents a primary storage system, and the storagesystem 301 represents a ternary storage system. In addition, A-Vol 310,B-Vol 320, C-Vol 330, and D-Vol 340 correspond to A-Vol 30 a-h, B-Vol 31a-h, C-Vol 37 a-h, and D-Vol 38 a-h, respectively. Shadow copy links 400and 410 and a remote copy link 420 correspond to the shadow copy links40 a-h and 47 a-h and the remote copy link 57 a-h, respectively.

FIG. 5 shows a process 202 performed by the control program 12 duringthe PiT remote copy and PiT shadow copy processes according to oneembodiment of the present invention. The process 202 is explained withreference to FIGS. 4 and 5. At step 210, the control program attempts toestablish a first shadow copy pair at the primary storage system 300between the A-Vol 310 and B-Vol 320 as indicated by the link 400 and asecond shadow copy pair at the ternary storage system 301 between theC-Vol 330 and the D-Vol 340 as indicated by the link 410. The above stepis reflected in the state (1) of FIG. 4. At the state (1), the C-Vols330 has a time consistent image at t0 of the ternary storage 301. TheD-Vol 340 does not yet have a time consistent image since the shadowcopy is underway. The A-Vol receives updates from the host, and theupdates are copied to B-Vol, so the B-Vols are not yet time consistentacross a consistency group.

In one implementation, the control program of a host in each of the sitehandles the shadow copy process at its own site. In this case, thecontrol programs communicate with each other through communication linkssuch as Local Area Network. In another implementation, a single controlprogram, e.g., the one associated with the primary site, controls theshadow copies in both the primary and ternary sites. In this case, thecontrol program sends commands to the ternary sites through remote copylinks.

At step 211, the control program executes a PiT freeze function to theshadow copy pairs 400 of A-Vols 310 to B-Vols 320. The state (2) of FIG.4 indicates the PiT freeze by placing a bar on the link 400.

At step 212, the control program executes a PiT query function to thepair 400 of A-Vol 310 to B-Vol 320 in the consistency group and checksits status. In one embodiment, a consistency group includes a pluralityof pairs of A-Vols 310 and B-VOL 320. The consistency group indicates agroup in which all volumes should be kept in consistency. This stepcontinues until all the pairs in the group indicate as having beenplaced in the Freeze Mode. This indicates that the B-Vols 320 have timeconsistent image at t1. The C-Vols 330 continues to have the timeconsistent image at t0.

At step 213, the program checks the shadow copy status of the shadowcopy pairs 410 in the consistency group in the ternary storage system310. This step is repeated until all the pairs in the group indicate asbeing placed on the Duplex Mode, whereby the D-Vols 340 are identical tothe C-Vols 330. In one implementation, there is no update on C-Volsduring this step.

At step 214, if all pairs are in the Duplex Mode, the program terminatesthe shadow copy process in the ternary storage system and suspends allpairs of C-Vols 330 to D-Vols 340. As a result, the D-Vols 340 and theC-Vols 330 are provided with time consistent image with respect to timet0. The state (3) of FIG. 4 illustrates this situation.

At step 215, once the pairs of C-Vols 330 to D-Vols 340 have beensuspended, the control program establishes remote copy links 420 betweenB-Vols 320 and C-Vols 330, thereby forming a plurality of pairs ofB-Vols and C-Vols. The remote copy links 420 correspond to the links 57a-h shown in FIG. 1. The D-Vols 340 have time consistent image at t0 inthe ternary storage 301. The C-Vols 340 serve as targets for the remotecopy from the B-Vols. Accordingly the C-Vols no longer have the timeconsistent image. Therefore, the earlier shadow copy to the D-Vol 340ensures that the time consistent image at the ternary storage site ismaintained. The state (4) of FIG. 4 illustrates the situation above.

At step 216, the control program checks the remote copy status of allpairs 420 of B-Vols 320 and C-Vols 330 in the consistency group. Thisstep continues until all the pairs in the group are placed in the DuplexMode that indicates C-Vols 330 are identical to B-Vols 320. In oneimplementation, there is no update on B-Vols during this step.

If all pairs are in the Duplex Mode, the program terminates the remotecopy process and suspends all pairs of B-Vols 320 and C-Vols 330 (step217). As a result, C-Vols 330 have the time consistent imagecorresponding to the data content of B-Vols 320 with respect to the timet1. This is indicated by the state (5). From the state (5), the programreturns to the state (I) and repeats the steps above. The time that ittakes to start at the state (I) and return back to the state (1) isreferred to as a cycle time. The states (1) to (5) define a copy cycle(or first copy cycle). Accordingly, subsequent states (1) to (5) definenext copy cycle (or second copy cycle). The first copy cycle is aprevious copy cycle of the second cycle.

The cycle time also indicates an approximate time to needed to make aPiT copy of the production data in the ternary storage 301, which ist1-t0. The cycle time should be reduced to a minimum to keep a freshcopy of the production data in the ternary site 3.

In order to reduce the cycle time, the state (2) including step 211 and212, and the state (3) including step 213 and 214 may be processed inparallel since there is no relation between these two states andprocesses. Under this embodiment, the program checks if both of thestates (2) and (3) are completed and prior to proceeding to the state(4).

Also, in order to eliminate unnecessary overheads, the process timerequired for the state (2) and the process time required for the state(3) preferably should be substantially the same. For this purpose, thecontrol program should efficiently determine point in times with PiTFreeze function as estimating the time required for the shadow copy fromC-Vols 330 to D-Vols 340.

One example of the method is provided as follows. At the state (2) ofevery cycle, the program calculates the amount of data to be shadowcopied from A-Vols 310 to B-Vols 320. This amount of data would beshadow copied from C-Vols 330 to D-Vols 340 in the next cycle.

Before the state (2) of the next cycle, the program estimates the timerequired for the shadow copy from C-Vols 330 to D-Vols 340 consideringthe amount of data copied from A-Vols 310 to B-Vols 320 in the previouscycle. Then, the program defines the point in time with PiT Freezefunction based on the estimated time.

FIG. 6 illustrates an information processing system 500′ according toanother embodiment of the present invention. In the system 500′, theprimary storage system resides in the site 2 x. That is, the productionsystem copies data to the secondary site 1 x that in turn copies data tothe ternary site 3. Accordingly, the direction of the remote copy links51 a-h between the primary site 2 x and the secondary site 1 x isdifferent from the first embodiment. The shadow copy processes in thestorage systems 20 a-h and 27 a-h are substantially the same asdescribed above. The remote copy between the storage system 20 a-h andthe storage system 27 a-h is also substantially the same as before.

However, the role of the storage systems 24 a-h in the primary site 2 xis different from the previous embodiment since the system 24 a-h servesas a production system rather than a back-up system. Also, the A-Vols 30a-h are now the targets of the synchronous remote copy links 51 a-h fromthe P-Vols 34 a-h. In this case, the synchronous remote copy links 51a-h must pass timestamps 16 a-h with IOs. In another embodiment, thesynchronous remote copy links 51 a-h must pass dummy IOs. The dummy IOsare issued to keep time consistency among the storage systems when allIOs to storage systems are frozen. The process detects the dummy IOsinstead of the particular timestamps when it operates PiT Freeze.

FIG. 7 shows a block diagram an information processing system 504according to one embodiment of the present invention. The system 504includes a primary storage system 60 and a plurality of distributedstorage systems 61 a-z. The primary storage system 60 has at least oneA-Vol 70 and a plurality of B-Vols 71 a-z. The B-Vols are targets ofshadow copy links 90 a-z from A-Vol 70. The distributed storage systems61 a-z have a plurality of C-Vols 80 a-z that are the targets of remotecopy links 91 a-z from B-Vols 71 a-z. The distributed storage systemsalso includes a plurality of D-Vols 81 a-z that are the targets ofshadow copy links 92 a-z from C-Vols 80 a-z.

The primary storage system 60 maintains production data. The distributedstorage systems 61 a-z maintains back-up data for use in a disasterrecovery and to share data among multiple sites distributedgeographically. To provide global data sharing, each distributed storagesystem 61 a-z includes time consistent data that is a point in time copyof the production data. Accordingly, a PiT shadow copy is performedbetween the A-Vol and the B-Vols in the system 60. Meanwhile, copies areperformed between the C-Vols and D-Vols in the distributed systems 61.Once the PiT shadow copy has been completed, a multiple remote copiesare performed between the B-Vols to the C-Vols, to provide C-Vols withthe consistent image of A-Vol.

The present invention has been described in terms of specificembodiments. As will be understood by those skilled in the art, theembodiments disclosed above may be modified, changed, or altered withoutdeparting from the scope of the present invention. Accordingly, thescope of the present invention is to be interpreted based on theappended claims.

1. A method for copying data to multiple remote sites, the methodcomprising: transmitting data from a first volume in a primary storagesystem to a back-up volume provided in a secondary storage system, theprimary storage system being located at a primary site and the secondarystorage system being located at a first remote site from the primarysite; copying the data from the first volume in the primary storagesystem to a plurality of second volumes in the primary storage system,wherein for each second volume a point in time (PiT) of said each secondvolume is used as a reference point of time for the copying, said eachsecond volume being provided with a first time consistent image of thefirst volume with respect to its reference point of time; and for eachsecond volume transferring the data therefrom to a corresponding thirdvolume in a ternary storage system at a second remote site, thecorresponding third volume being provided with a second time consistentimage of said each second volume which is substantially the same as thefirst time consistent image provided in said each second volume.
 2. Themethod of claim 1, further comprising: for each third volume copying thedata therefrom to a corresponding fourth volume in the ternary storagesystem of said each third volume, the corresponding fourth volume beingprovided with a third time consistent image corresponding to the secondtime consistent image provided in said each third volume, the third timeconsistent image being substantially the same as the first timeconsistent image provided in the second volume corresponding to saideach third volume.
 3. The method of claim 1, wherein the transmittingstep involves a synchronous remote copying method, and the transferringstep involves an asynchronous remote copying method.
 4. The method ofclaim 1, wherein the first and second time consistent images aresubstantially the same.
 5. The method of claim 1, further comprising:receiving a plurality of data write requests at the primary storagesystem from a primary host, each of the data write requests having atimestamp and data associated with that write request; and storing thedata write requests in the primary storage system.
 6. The method ofclaim 5, wherein the copying step includes: retrieving a first timestampassociated with first data; and for each of the second volumes shadowcopying the first data from the first volume to said each second volumeas second data if the first timestamp indicates a time that is prior tothe reference point of time of said each second volume.
 7. The method ofclaim 6, further comprising: retrieving a second timestamp associatedwith the second data; and not copying the second data to said eachsecond volume if the second timestamp indicates a time that issubsequent to the reference point of time of said each second volume. 8.The method of claim 6, further comprising: suspending the copying stepif all data stored in the first volume having timestamps that precedethe reference point of time of said each second volume are shadow copiedto said each second volume.
 9. The method of claim 8, furthercomprising: setting a freeze mode for the first and second volumes. 10.The method of claim 9, further comprising: determining whether or notthe first and second volumes are placed in the freeze mode; and checkingthe status of the second and third volumes to determine if the secondtime consistent images of the second volumes have been copied entirelyto the corresponding third volumes.
 11. The method of claim 10, furthercomprising: suspending the transferring step for each correspondingthird volume upon determining that said each corresponding third volumehas been provided with a time consistent image of its corresponding tothe second time consistent image.
 12. The method of claim 1, furthercomprising: for each third volume copying the data therefrom to acorresponding fourth volume in the ternary storage system of said eachthird volume, the corresponding fourth volume being provided with athird time consistent image corresponding to the second time consistentimage provided in said each third volume; and estimating time requiredfor providing the corresponding fourth volume with the third timeconsistent image using information relating to an amount of data copiedfrom the first volume to the second volume corresponding to said eachthird volume in a previous copy cycle.
 13. The method of claim 12,wherein the amount of data corresponds to the data copied from the firstvolume to the second volume corresponding to said each third volume toprovide the second volume with the second time consistent image.
 14. Themethod of claim 12, wherein the information relating to the amount ofdata copied from the first volume to the second volume corresponding tosaid each third volume corresponds to a copy time needed to provide thesecond volume with the second time consistent image.
 15. A method forcopying data to a remote site, the method comprising: copying data froma first volume to a plurality of second volumes to provide each thesecond volumes with a first time consistent image with respect to afirst given time, the first and second volumes being provided in a firststorage system; and transferring the data from each second volume to acorresponding third volume to provide the corresponding third volumewith a corresponding second time consistent image with respect to asecond given time, the corresponding third volume being provided in asecond storage system that is located at least 10 miles from the firststorage system.
 16. The method of claim 15, further comprising:transmitting the data stored in the first volume to a back-up volumeprovided in a third storage system, wherein the transmitting stepinvolves a synchronous remote copying method and the transferring stepinvolves an asynchronous remote copying method.
 17. The method of claim15, further comprising: receiving the data at the first storage systemfrom a third storage system, wherein the receiving step involves asynchronous remote copying method and the transferring step involves anasynchronous remote copying method, wherein the first and second storagesystems are storage sub-systems, wherein the first given time and thesecond given time are the same.
 18. (canceled)
 19. A computer system,comprising: a timer to provide a timestamp to data requests; aninterface configured to form a communication link with a first storagesub-system; and a computer storage medium including: code for initiatingcopying of data from a first volume to a plurality of second volumes toprovide each of the second volumes with a first time consistent imagewith respect to a first given time, the first and second volumes beingprovided in a first storage sub-system, and code for initiatingtransferring of the data from each second volume to a correspondingthird volume to provide the corresponding third volume with a secondtime consistent image with respect to a second given time, thecorresponding third volume being provided in a second storage systemthat is located at least 10 miles from the first storage sub-system. 20.The computer system of claim 19, wherein the computer system is a hostcoupled to the first storage sub-system or a storage system including aplurality of storage sub-systems provided at multiple sites.
 21. Acomputer readable medium for use in a storage system, the mediumcomprising: code for copying data from a first volume to a plurality ofsecond volumes to provide each of the second volumes with a first timeconsistent image with respect to a first given time, the first andsecond volumes being provided in a first storage system; and code fortransferring the data from each second volume to a corresponding thirdvolume to provide the corresponding third volume with a second timeconsistent image with respect to a second given time, the correspondingthird volume being provided in a second storage system that is locatedat least 10 miles from the first storage system.
 22. The computerreadable medium of claim 21, wherein the medium is provided in a host orstorage sub-system.
 23. A method for copying a volume in a storagesystem, the method comprising: checking a first timestamp of a firstdata to be copied from a first volume to a plurality of second volumes;and for each second volume: copying the first data to said each secondvolume if the first timestamp is prior to a given reference point;checking a second timestamp of a second data to be copied from the firstvolume to said each second volume; suspending the copy operation if thesecond timestamp is after the reference point; and placing said eachsecond volume in a Freeze mode to indicate that said each second volumeincludes a point in time (PiT) copy of the first volume.
 24. The methodof claim 23, wherein the first and second volumes are defined in thesame storage disk array, wherein the first volume receives a writerequest from a host while data are being copied from the first volume tothe second volumes.
 25. (canceled)
 26. A method for providing a point intime remote copy, the method comprising: checking a first timestamp of afirst data to be copied from a first volume to each of second and thirdvolumes, the first, second, and third volumes being provided in the samestorage system; copying the first data to the second and third volumesif the first timestamp is prior to a given reference point; remotecopying the first data from the second and third volumes to fourth andfifth volumes, respectively; transmitting the first data from the fourthand fifth volumes to six and seventh volumes, respectively, wherein thefourth and six volumes are provided in a first remote storage system andthe fifth and seventh volumes are provided in a second remote storagesystem; checking a second timestamp of a second data to be copied fromthe first volume to the second and third volumes; and placing the secondand third volumes in a Freeze mode to indicate that the second volumeincludes a point in time (PiT) copy of the first volume if the secondtimestamp of the second data to be copied from the first volume to thesecond and third volumes is after the reference point.
 27. The method ofclaim 1 wherein each third volume is in a ternary storage systemdifferent from ternary storage systems of other third volumes.